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Introduction 



In this paper a domain QQC n is a connected open set. We let 0{Q) de- 
note the algebra of holomorphic functions on Vl. Also we let Aut(fi) be the 
group (under composition of mappings) of biholomorphic self-maps of Q. The 
standard topology on Aut(O) is that of uniform convergence on compact sets 
(equivalently, the compact-open topology). 

We shall use the following notation: D denotes the unit disc in the com- 
plex plane. We let D 2 = D x D denote the bidisc, and D n = DxDx---xD 
the polydisc in C n . The symbol B = B n is the unit ball in C n . 

Certainly domains with transitive automorphism group are of some in- 
terest. But they are relatively few in number (see the classification theory of 
Cartan, as described in [HEL]). A very natural and compelling alternative is 
to study domains with noncompact automorphism group. A bounded domain 
fl has noncompact automorphism group if there is a sequence ipj G Aut(f2) 
such that no subsequence converges uniformly on compact sets to another 
automorphism. Obversely, the automorphism group is compact if every se- 
quence {<fj} in Aut(f2) has a subsequence that converges uniformly on com- 
pact sets to another automorphism. In this regard, the following result of H. 
Cartan is central and useful (see [NAR]): 

Theorem 0.1 Let f2BC n be a bounded domain. Then Q has noncompact 
automorphism group if and only if there are a point P G dQ and a point 
IgU and automorphisms ipj G Aut(Q) such that lim^oo <Pj(X) = P. 

A point P G dQ is called a boundary orbit accumulation point if there is a 
point XgH and a sequence ipj G Aut(f2) such that lim^oo (pj (X) = P. Of 
special interest is the case when there is a single automorphism ip such that 
lim^oo ^(X) = P. (Here denotes the composition of tp with itself j times 
when j = 0, 1, 2, ... ; also, if j < 0, then ^ denotes the composition of ip~ l 
with itself \j\ times.) In this latter circumstance we call P a special boundary 
orbit accumulation point. It is not clear when an arbitrary boundary orbit 
accumulation point is a special boundary orbit accumulation point. 

We shall use Section 1 to collect some simple, preliminary results that 
have independent interest. 
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1 Background Results 



The result that inspires the present work comes from [GRK1]: 

Theorem 1.1 Let J18C n be a smoothly bounded domain. Suppose that 
P G dQ is a boundary orbit accumulation point. Then P is a point of Levi 
pseudoconvexity. 

There is an analogous result for domains without smooth boundary (and 
in which the conclusion involves Hartogs pseudoconvexity). But we shall 
have no use for it in the present paper. See [GRK1] for the details. 

We first give an example to emphasize that, even though the boundary 
orbit accumulation point is pseudoconvex, nearby points need not be. 

EXAMPLE 1.2 Let B8C n be the unit ball with defining function p(z) = 
\z\ 2 — 1 (see [KRAI] for the concept of definining function). Let be a 
function on C n with these properties: 

(a) is real- valued and < </>(z) < 1/10 for all z G C n . 

(b) is radial about the point (i, 0). 

(c) supp (j)W{(i,0), 1/10). 

(d) 4>{z) = 1/10 for \z-(i,0)\ < 1/20. 
Set 

Q' = {z G C 2 : -1 + \z\ 2 + <j>(z) < 0} 

and 

oo 

^= n $ i/ 2 ( fi/ ), 

j=—oo 

where 

( Z!-a y/1 - \a\ 2 z 2 \ 

$a(2l, Z2) = -, — , = , 

V 1 — azi 1 — az\ j 

any a G C, \a\ < 1. It is easy to check, by direct calculation, that $ a is an 
automorphism of the unit ball BQC n . And the domain Q will be the unit 
ball with infinitely many strongly pseudoconcave dents that accumulate at 
the points (1,0) and (—1,0). 
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Now it is plain that the point (1,0) G dQ is a boundary orbit accumulation 
point. In fact we may let X = (0, 0) and (pj(z) = $y 2 { z ) for j = 1,2,.... 
So (1,0) G dQ is certainly pseudoconvex. Notice that, at points along the 
normal line through (i,0), —1 + \z\ 2 + (f>(z) is negative when z is at least 
1/5 units from the boundary of the ball B. And —1 + \z\ 2 + <f>(z) is positive 
at (i, 0). So there must be an intermediate point z on this line segment — a 
point inside the unit ball 5 — where — 1 + \ z\ 2 + <p(z) vanishes. It follows that 
z is a boundary point of Hence $^ 2 (F) is a boundary point for each j. 

The boundary points $^ 2 (i) will be strictly pseudo concave. 

It must be noted that, in the last example, Q does not have smooth 
boundary. In fact, at the boundary points (1,0), (—1,0), the boundary is 
only Lipschitz. 

Of course the disc DQC has noncompact automorphism group. Let 

¥>a(C) 



1-aC 

for a G C, |a| < 1. Then the automorphisms 

C-(i-Vi) 



i - (i - Vi)C 



:j'GZ 



are a sequence of automorphisms of D that have no subsequence converg- 
ing to an automorphism. Indeed, any subsequence either converges to the 
constant function 1 or the constant function —1. It is a fact — see [KRA2]- 
that any domain in C having C 1 boundary and noncompact automorphism 
group must be conformally equivalent to the disc. This is true without any 
topological hypotheses on the domain! In C n , the first correct theorem of 
this nature — due to Bun Wong [WON] and Rosay [ROS] — is that any C 2 
bounded domain in C n with a boundary orbit accumulation point that is 
strongly pseudoconvex must be biholomorphic to the unit ball B. It is not 
known in general which smoothly bounded domains have noncompact auto- 
morphism group. Certainly there are finite type domains with non-compact 
automorphism group — see [BEP1], [BEP2]. More on this matter in what 
follows. 

In this paper we concentrate mainly on bounded domains. But we shall 
make a few remarks right now about unbounded domains. 
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EXAMPLE 1.3 In the complex plane C, there are unbounded domains 
with noncompact automorphism group that are not the disc. The simplest 
example is when the domain Q is the entire complex plane C. The punctured 
plane also has this property. 

2 New Results 

The statement of Theorem 1.1 makes it desirable to have a formulation purely 
in terms of the intrinsic, invariant geometry of the domain. For instance, 
one might hope to be able to say something about the completeness of the 
Kobayashi metric at a boundary orbit accumulation point. Unfortunately, 
the following example dashes that hope: 

EXAMPLE 2.1 Let B8C 2 be the unit ball with defmining function p(z) = 
\z\ 2 — 1. Let be a function on C n with these properties: 

(a) is real- valued and < <j>{z) < 1/10 for all z e C n . 

(b) is radial about the point (i, 0). 

(c) supp <j)W((i,0), 1/10). 

(d) 4>{z) = 1/10 for \z-(i,0)\ < 1/20. 
Set 

n f = {z EC 2 : -1 + \z\ 2 + 4>{z) < 0} 

and 

n= P| $ a (Q'), 

-Ka<l 

where 

( z 1 -a y/1 - \a\ 2 z 2 \ 
\ 1 — azi 1 — azi I 

any a e C, \a\ < 1. Then Q is the unit ball with a groove stretching from 
(— 1, 0) to (1, 0). This new domain is strongly pseudoconcave along an entire 
curve from (—1,0) to (1,0). Of course the point (1,0) is still a boundary 
orbit accumulation point. Indeed the automorphisms $ a , — 1 < a < 0, send 
(0,0) to (1,0). And, along the curve 7(f) = (t,0), < t < 1, the Kobayashi 
distance to the boundary point is infinite. 
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Now write $ a ( z ) — (v 9 ^- 2 )' Pl( z ))- Let the point z be as at the end of 
Example 1.2. We consider, in the same domain fl' as above, the curve 

([1 - (1 - t) 4 ]<^ t (F),0) , 0<t< 1. 

Then this curve terminates at (—1,0) and is tangent to the boundary to 
fourth order at that point. For Q e dfl, let vq denote the outward Euclidean 
unit normal vector at Q. If we take advantage of the estimate in [KRA3], 
to the effect that, near a strongly pseudoconcave boundary point Q, a point 
Q* = Q — 5uq satisfies the estimate 

and moreover, in the tangential direction, 

i$(QV)«C, 

then it is easy to see that the distance to the boundary point (—1,0) along 
the curve 7 is finite. Therefore, at least on a domain with Lipschitz boundary, 
it is not the case that a boundary orbit accumulation point will be a point 
at which the Kobayashi metric is complete. 

On the positive side, we can prove the following result: 

Proposition 2.2 Let f2BC 2 be a smoothly hounded domain and P £ dil 
a boundary orbit accumulation point that is of hnite type. Then it is not 
possible for Q to have Levi pseudoconcave boundary points. 

Proof: Assume that P is the limit of (pj(X) for some point X e fl and some 
automorphisms ipj. Since P is of finite type and the complex dimension is 2, 
we may use results of [CAT] and [ALA] to see that curves in Q terminating 
at P must have infinite length in the Kobayashi metric. 

If Q is a strongly pseudoconcave boundary point, obviously distinct from 
P, then we know from the estimates in [KRA3] that the Kobayashi distance 
of Q to X is finite. Then the tpj applied to Q must move Q into the interior 
of Q (because X is being pushed infinitely far towards P). Hence if is not an 
automorphism. That is a contradiction. We conclude that Q cannot exist. □ 

A well-known conjecture in the subject says this: 
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The Greene-Krantz Conjecture: Let f2 be a smoothly bounded 
domain in C n . If P G dQ is a boundary orbit accumulation point, 
then P is a point of finite type in the sense of Kohn/D'Angelo/Catlin. 

This conjecture has not been established in full generality. But results in 
[KIM] and [KIK1] support the conjecture. Now we have 

Proposition 2.3 Let fifiC 2 he a smoothly hounded, pseudoconvex domain. 
Let P G dQ be a boundary orbit accumulation point. Assume that the 
Greene-Krantz conjecture is true. Then any path ending at P will have 
infinite length in the Kobayashi metric. 

Proof: This result is almost obvious. For the hypothesis implies that P is 
of finite type. And now the estimates on the Kobayashi metric in [CAT], 
together with the calculations in [ALA], give the result about infinite length 
of paths. □ 



3 A Boundary Orbit Accumulation Point Char- 
acterization of Domains 

In [ROS] and [WON], for instance, it is shown that if a bounded domain 
has a strongly pseudoconvex boundary orbit accumulation point then that 
domain must be biholomorphic to the unit ball in C n . Put in other words, 
if two distinct bounded domains have boundary orbit accumulation points, 
and if those boundary orbit accumulation points are both strongly pseudo- 
convex, then the two domains must be biholomorphic (since they are both 
biholomorphic to the ball). 

One might more generally formulate this question 

Suppose that fli and f2 2 are two bounded domains in C n . Assume 
that Qi has boundary orbit accumulation point Pi and Q 2 has 
boundary orbit accumulation point Pi- If -Pi and P 2 have the 
same Levi geometry, may we conclude that Q± is biholomorphic 
to tt 2 ? 

I do not know the full answer to this question at this time. However, the 
following partial answer may be proved using known techniques: 
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Proposition 3.1 Let Q 2 be bounded domains in C n . Let Pi G dQi and 

P 2 G dQ 2 each be boundary orbit accumulation points. Assume that dQj 
is smooth near Pj, j = 1,2. Suppose that each Pj is of finite type in the 
sense of Kohn/Catlin/D'Angelo and is also a peak point. Finally assume 
that there is a neighborhood JJ\ of P\ and a neighborhood U 2 of P 2 and a 
biholomorphic mapping 

$:[/inn 1 ^f/ 2 nn 2 

such that (i) $ continues to a diffeomorphism of d£li n XJ\ to dQ 2 H U 2 , (ii) 

$ (Pi) = P2. 

Then Qi is biholomorphic to Q 2 . 

Proof: Choose a point Xi G Qi and automorphisms (fij of £l\ so that 
ipj(Xi) — > P\. Likewise choose a point X 2 G Vt 2 and automorphisms ipj 
of ^2 such that ipj(X 2 ) — > P 2 . A standard argument (see [KRAI, Chapter 
11) shows that, for any compact set K C Q ±J (fij(z) converges to Pi uniformly 
for z G K. A similar statement holds for Q 2 - 

Let Kbea large compact set inside Q 1 . Choose j so large that (pj(K)QUin 
Qi. Likewise let L be a large compact set inside ^2- Choose k so large that 
ip k (L)QU 2 n Q 2 - Let e > be small and set U[ — {z G XJ\ : dist(z, c Ui > e}. 
Similarly set U 2 = {z G U 2 : dist(z, C U 2 > e}. By shrinking e if necessary, we 
may assume that Q(ipj(K))QU 2 . By enlarging L if necessary, we may suppose 
that (p k (L) D Ul 

Now consider (ipk) -1 $ fj- This will be a univalent holomorphic map- 
ping that takes KQQi to L. And the mapping is invertible. We may similarly 
assume that the inverse mapping takes L to K. The set of all such mappings, 
as K exhausts Oi and L exhausts Q 2 forms a normal family. And we may 
extract a convergent subsequence that converges to a biholomorphic mapping 
of f2i to il 2 . That is the result that we seek. □ 

There are a number of different approaches to the classical Bun Wong/Rosay 
theorem. Useful references are [WON], [ROS], [KIK2], [KIK3], [KIK4], [GKK]. 
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4 Concluding Remarks 



In the paper [BEP1] and subsequent works, Bedford an Pinchuk prove results 
of the following type: 

Theorem: Let Q be a pseudoconconvex domain in C 2 with real 
analytic boundary. Suppose that Q has noncompact automor- 
phism group. Then Q must be biholomorphic to a complex ellip- 
soid of the form 

E m = {(z 1 ,z 2 )EC 2 :\z 1 \ 2 + \z 2 \ 2m <l} 

for m — 1,2, 

In a private communication, David Catlin pointed out that the hypothesis 
of this theorem may be reduced to "finite type domain" in C 2 . In later papers 
Bedford and Pinchuk produced analogous results in C n . 

But it must be pointed out that, in higher dimensions, we cannot hope 
for a conclusion as simple as "the domain must be a complex ellipsoid." For 
consider the domain 

Q* = {(z u z 2 , z 3 ) e C 3 : N 2 + (\z 2 \ 2 + N 2 ) 2 < 1} . 

It has automorphisms of the form 

( Zl -a - \a\ 2 z 2 {/l-\a\ 2 z 3 \ 
Vayzi, z 2 , z 3 - - — — _ , — , 
\ 1 — azi v 1 — azi a/1 — az 1 I 

for a G C, \a\ < 1. If we let a take the values 1 — for j — 1, 2, . . . then we 
see immediately that Q* has noncompact automorphism group. And Q* is 
not an ellipsoid in the most direct sense. It has been conjectured by Catlin 
and others (see [KRA4] for the details) that the correct conclusion in higher 
dimensions is that the defining function of the domain should satisfy a certain 
homogeneity condition. 

It has been noted that the Greene-Krantz conjecture asserts that a bound- 
ary orbit accumulation point must be of finite type. There is some evidence to 
support the conjecture — see, for instance, [KIM], [KIK1]. If it turns out to be 
true, then the Bedford/Pinchuk theorem cited above can be streamlined to 
say that a smoothly bounded domain in C 2 with noncompact automorphism 
group must be an ellipsiond. 
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It is certainly a matter of some interest to understand the nature of 
boundary orbit accumulation points. We know that they must be pseudo- 
convex, and the Greene-Krantz conjecture posits even more specific informa- 
tion about these points. Another subject of some study is boundary orbit 
accumulation sets — see, for instance, [ISK] and [KRA5]. Much more can in 
principle be said about these sets. 

Automorphism groups are in some sense an invariant that is a substitute 
for the lack in several complex variables of a uniformization theorem or a 
Riemann mapping theorem. It is in our best interest to develop their prop- 
erties so that they can be used effectively to study and classify domains up 
to biholomorphic equivalence. 

We hope to study these matters further in future papers. 
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